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FOREWORD 


The  Computer-Based  Educational  Technology  team  of  the  U.S.  Army  Re¬ 
search  Institute  for  the  Behavioral  and  Social  Sciences  (ARI)  performs 
research  and  development  in  areas  of  educational  technology  with  applica¬ 
bility  to  military  education  and  training.  Of  interest  are  instructional 
delivery  systems  with  a  special  focus  on  the  development  of  accompanying 
courseware  in  the  most  efficient  and  cost  effective  manner  possible. 

The  development  and  implementation  of  such  systems  address  the  problem 
of  training  individuals  who  must  produce  good  courseware  in  a  reasonable 
time,  at  an  acceptable  cost. 

This  Technical  Report  provides  an  annotated  bibliography  of  mate¬ 
rials  available  to  support  instructional  development.  It  also  identifies 
those  areas  in  which  further  research  and  development  is  required. 

Appreciation  is  extended  to  Dr.  Richard  Braby,  Department  of  the 
Navy,  Training  Analysis  and  Evaluation  Groupi  Dr.  John  Ellis,  Navy  Per¬ 
sonnel  Research  and  Development  Center;  and  Dr.  J.  Dexter  Fletcher,  De¬ 
fense  Advanced  Research  Projects  Agency;  for  reviewing  an  earlier  draft 
of  this  report. 

The  research  effort  is  responsive  to  the  requirements  of  RDT&E 
Project  2Q762722A764,  Educational  Technology  and  Simulation  Technical 
Area  of  the  FY  78  ARI  Work  Program. 
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AN  ANNOTATED  BIBLIOGRAPHY  FOR  INSTRUCTIONAL  SYSTEMS  DEVELOPMENT 


BRIEF 


Requirement: 

To  determine  the  state-of-the-art  in  instructional  systems  develop¬ 
ment  and  to  specify  critical  research  gaps* 


Procedure: 

A  review  of  the  instructional  development  literature  was  conducted. 
An  annotated  bibliography  was  produced  which  organised  references  accord¬ 
ing  to  the  steps  of  the  Interservice  Procedures  for  Instructional  Systenu 
Development  Model  (ISO,  TRADOC  Pamphlet  350-30).  Each  reference  was 
classified  as  (1)  an  authoring  aid,  i. e. ,  a  "how-to-do-it"  manual  which 
directly  guides  the  author  through  an  activity  identified  by  the  ISD 
model,  (2)  a  procedure,  i.e. ,  a  listing  of  the  activities  involved  in 
ISD  or  (3)  a  technique,  i.e.,  a  methodology  for  accomplishing  an  activ¬ 
ity  but  lacking  sufficient  detail  to  be  a  procedure. 


Findings: 

(1)  In  Phase  1 -ANALYZE,  authoring  aids  are  available  for  Select 
Tasks/Functions;  Construct  Job  Performance  Measures:  Analyse  Existing 
Courses;  and  Select  Instructional  Setting.  Authoring  aids  are  unavail¬ 
able  for  Analyze  Job. 

(2)  In  Phase  II-DESIGN,  authoring  aids  are  available  for  Develop 
Objectives  and  Develop  Tests.  Authoring  aids  are  unavailable  for  De¬ 
scribe  Entry  Behavior  and  Determine  Sequence  and  Structure. 

(3)  In  Phase  II I- DEVELOP,  authoring  aids  are  available  for  all 
activities  involving  print  as  a  medium.  However,  authoring  aids  are 
unavailable  for  computer-based  instructional  materials. 

(4)  In  Phase  IV- IMPLEMENT,  authoring  aids  are  unavailable. 

(5)  In  Phase  V- CONTROL,  authoring  aids  are  unavailable. 


vii 


Utilization  of  Findings: 


The  identification  of  aids,  procedures,  or  techniques  will  be  of 
immediate  use  to  those  involved  in  instructional  systems  development. 
In  addition,  these  findings  will  form  one  basis  for  a  future  research 
and  development  plan. 
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AN  ANNOTATED  BIBLIOGRAPHY  FOR  INSTRUCTIONAL 
SYSTEMS  DEVELOPMENT 


INTRODUCTION 

The  need  for  the  Instructional  Systems  Development  (ISD)  was  iden¬ 
tified  in  field  tests  of  individualized  instructional  training  systems. 

It  was  difficult  to  bring  individuals  who  used  these  systems  to  the 
level  of  competence  needed  to  produce  good  curriculum  materials  in  a 
reasonable  time  at  an  acceptable  cost.  Similar  difficulties  have  been 
noted  with  other  systems  employing  computer-based  instruction.  Ibis 
problem  is  especially  acute  in  the  Army  because  of  the  high  turnover 
of  personnel.  Further,  since  future  Army  training  and  education  sys¬ 
tems  will  be  individualized,  the  need  for  this  technology  is  likely 
to  intensify. 

Currently  the  instructional  systems  development  process  is  time  con¬ 
suming.  It  is  not  unusual  to  find  that  it  takes  one  hundred  or  more 
hours  to  develop  and  evaluate  one  instructional  hour  of  self-paced  mate¬ 
rial.  For  example,  Orlansky  { 1979)  provides  evidence  for  variability  and 
length  of  time  among  authors  to  produce  instructional  material.  His  data, 
which  summarize  all  available  DoD  studies  in  computer-based  instruction, 
indicates  that  authors  vary  by  a  factor  of  10  among  themselves.  In  addi¬ 
tion,  several  sources  (e.g.,  Avner,  1979)  have  indicated  that  the  training 
process  takes  at  least  six  months  before  authors  can  be  expected  to  pro¬ 
duce  useful  materials  and  a  year  before  full  production. 

The  Army's  first  attempt  at  instructional  systems  development  was 
called  systems  engineering  (United  States  Continental  Army  Command  Regu¬ 
lation  350-100-1,  February  1968).  The  most  recent  effort  of  the  Army 
to  provide  standardized  procedures  for  the  development  and  conduct  of 
training  is  the  Interservice  Procedures  for  Instructional  Systems  Devel¬ 
opment  Model  (ISD,  1975).  The  model  provides  for  the  assessment  of  train¬ 
ing  needs;  the  design,  development  and  implementation  of  instruction  and 
the  assessment  of  instructional  quality  through  a  five  phase  process  (see 
Figure  1).  The  first  phase  is  ANALYZE,  which  provides  guidance  on  task 
analysis,  the  selection  of  tasks  to  be  trained,  the  development  of  measures 
of  job  performance  and  the  selection  of  the  appropriate  environment  for 
training.  DESIGN  is  the  second  phase  which  establishes  the  objectives, 
test  items,  and  sequence  of  the  course.  The  entry  behavior  or  skills 
the  trainee  arrives  with  are  also  noted  during  this  phase.  The  third 
phase  is  DEVELOPMENT  during  which  the  instructional  materials  are  created. 
Existing  materials  are  examined  so  that  new  course  materials  are  devised 
only  when  appropriate  others  do  not  exist.  A  plan  specifying  all  activi¬ 
ties  of  the  learner  is  made  as  well  as  a  plan  for  pilot  testing  the  newly 
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developed  materials.  During  IMPLEMENTATION,  Phase  IV,  the  plan  of  Phase 
III  is  activated  with  particular  attention  to  the  personnel  needed  to 
accomplish  the  plan.  The  quality  of  the  instruction  is  assessed  during 
Phase  V  CONTROL  in  terms  of  new  skills  acquired  by  the  trainee  and  the 
fulfilled  needs  of  the  Command.  As  is  shown  in  Figure  1,  the  five  phases 
(identified  by  Roman  numerals)  have  been  elaborated  upon  in  19  blocks, 
(Arabic  numbers),  each  with  its  own  inputs,  outputs,  and  feedback  loop 
following  the  requirements  of  a  systems  engineering  approach.  A  brief 
example  on  notation  is  Block  II. 1  which  refers  to  the  second  phase-DESIGN 
and  first  block — Develop  Objectives. 

The  remainder  of  this  report  will  briefly  discuss  a  problem  observed 
with  the  implementation  of  the  ISD  model,  provide  an  annotated  biblio¬ 
graphy  addressing  the  problem  and  pose  directions  for  further  research 
and  development. 


PURPOSE 

The  activities  of  military  instructional  developers  have  been  speci¬ 
fied  by  a  TRADOC  Regulation  (Pamphlet  350-30)  which  is  formally  known  as 
the  Interservice  Procedures  for  Instructional  Systems  Development  Model 
(ISD).  This  document  provides  "procedures"  to  be  followed  when  devising 
a  course  of  instruction.  These  procedures  reflect  the  state-of-the-art 
when  the  model  was  devised.  Unfortunately  little  guidance  beyond  "what 
needs  to  be  done"  is  provided  by  the  ISD  model. 

The  present  report  was  designed  to  review  instructional  development 
resources  in  an  effort  to  provide  more  proceduralized  guidance  to  indi¬ 
viduals  tasked  with  implementing  the  ISD  model.  These  individuals  are 
referred  to  as  course  authors.  By  definition  authoring  materials  may 
take  the  form  of  research  studies,  textbooks,  handbooks,  computer  programs, 
audiovisual  sources,  or  technical  manuals  as  long  as  the  intention  of 
the  resource  is  to  guide  the  analysis,  design,  development,  implementa¬ 
tion,  or  control  of  instruction.  Several  topics  are  outside  the  boundary 
conditions  of  this  report.  These  include  research  and  development  related 
to  cost-effectiveness  (e.g. ,  Seidel  and  Wagner,  1979)  and  artificial  in¬ 
telligence  applications  in  instruction  (e.g.,  Gentner,  1979). 


METHOD 

The  present  research  was  intended  to  update  and  supplement  a  pre¬ 
vious  literature  review  and  annotated  bibliography  of  authoring  mate¬ 
rials  which  was  prepared  and  updated  by  Logan  (1977,  1979).  Many  of 
Logan's  conventions  were  adopted  so  that  this  report  could  be  used  in 
conjunction  with  his.  However,  this  document  includes  the  most  cogent 
of  Logan's  references  so  that  it  is  complete  in  itself  as  well.  The 
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bibliography  was  formed  over  the  course  of  four  activities:  search 
classify,  summarize,  and  assess. 


A  literature  search,  beginning  with  documents  published  in  1960, 
was  conducted  manually  through  the  library  of  the  U.S.  Army  Research 
Institute  for  the  Behavioral  and  Social  Sciences,  Psychological  Ab¬ 
stracts  and  abstracts  from  meetings  of  relevant  professional  organiza¬ 
tions.  Computer-based  data  searches  were  made  with  the  Defense  Documenta¬ 
tion  Center  ( DDC ) ,  the  National  Technical  Information  Center  (NTIS),  and 
the  Educational  Resources  Information  Center  (ERIC).  Some  of  the  de¬ 
scriptors  used  in  the  search  activities  were: 

Audiovisual  Communications  Media 
Behavioral  Objectives 
Computer-Assisted  Instruction 
Computer-Managed  Instruction 
Education 

Educational  Program  Evaluation 

Individualized  Instruction 

Instructional  Materials 

Job  Analysis 

Systems  Analysis 

Task  Analysis 

Test  Construction 

Testing 

Training 

Documents  were  also  made  available  through  personal  contacts  with  re¬ 
searchers  and  practitioners  knowledgeable  in  instructional  systems  devel¬ 
opment.  Copies  of  relevant  documents  were  made  on  paper  or  microfiche 
and  manually  stored. 

All  documents  were  then  classified  according  to  the  19  block  scheme 
indicated  by  the  ISD  model  (see  Figure  1).  In  some  cases  a  document  ap¬ 
plied  to  more  than  one  block.  In  those  cases  it  was  annotated  for  each 
block. 


Next,  a  short  summary  was  written  after  a  thorough  reading  of  each 
document.  If  a  document  was  summarized,  both  authors  felt  it  was  of 
value.  The  purpose  of  the  summary  statement  was  to  identify  the  docu¬ 
ment  as  an  authoring  aid,  procedure,  or  technique.  An  authoring  aid 
was  defined  as  a  "how  to  do  it"  manual  which  directly  guides  the  author 
through  an  activity  identified  by  the  ISD  model.  A  procedure  was  de¬ 
fined  as  a  listing  of  the  activities  involved  in  a  block  of  the  ISD 
model.  A  technique  was  defined  as  a  methodology  for  accomplishing  an 
activity  identified  by  the  ISD  model  but  lacking  sufficient  detail  to 
be  a  procedure.  Finally,  the  status  of  authoring  aids  and  procedures 
a.aiiaole  was  made  on  a  block  by  block  basis. 
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In  the  RESULTS  section  which  follows,  the  annotated  bibliography  is 
presented  with  a  summary  of  the  purpose  of  each  block.  The  references  are 
organized  by  block  and  then  alphabetically  within  block,  A  letter  pre¬ 
ceding  each  reference  indicates  whether  it  is  an  authoring  aid  (A),  pro¬ 
cedure  (P),  or  technique  (T).  A  reference  numbered  A  2,4.8  identifies 
an  authoring  aid  for  Phase  II-DESIGN,  Block  4-Determine  Sequence  and 
Structure  which  is  the  eighth  source.  The  notation  in  Arabic  numbers 
identifying  the  source  is  consistent  with  Logan  (1979), 

A  Status  section  for  each  block  indicates  the  availability  of  author¬ 
ing  aids  sufficient  to  guide  an  individual  through  all  activities  speci¬ 
fied  by  the  block  as  well  as  the  availability  of  relevant  procedures  and 
techniques  which  could  be  developed  into  authoring  aids.  A  figure  in¬ 
dicating  the  availability  of  authoring  aids,  procedures,  and  techniques 
by  phase  and  block  follows,  using  the  A,  T,  and  P  notation  established 
earlier. 

In  some  cases  the  status  of  a  block  may  have  indicated  procedures  but 
no  techniques.  It  can  be  assumed  that  if  a  procedure  exists  a  technique 
was  available  prior  to  its  development.  Educational  Research  Information 
Center  or  National  Technical  Information  Service  numbers  were  noted  when 
available. 

It  should  be  noted  once  again  that  the  main  intention  of  the  anno¬ 
tated  bibliography  was  to  identify  authoring  aids.  The  inclusion  of  pro¬ 
cedures  and  techniques  was  intended  to  direct  developmental  efforts 
toward  authoring  aids. 


5 


REFERENCES 


Avner,  R.  A.  Production  of  Computer-based  Instructional  Materials.  In 
H.  F.  O’Neil,  Jr.  (Ed.),  Issues  In  instructional  systems  develop¬ 
ment.  New  York:  Academic  Press,  1979. 

Gentner,  D.  R.  Toward  an  Intelligent  Computer  Tutor.  In  H.  F. 

O’Neil,  Jr.  (Ed.),  Procedures  for  instructional  systems  develop¬ 
ment.  New  York:  Academic  Press,  1979. 

Logan,  R.  S.  A  State  of  the  Art  Assessment  of  Instructional  Systems 

Development.  In  H.  F.  O'Neil,  Jr.  (Ed.),  Issues  in  instructional 
and  systems  development.  New  York:  Academic  Press,  1979. 

Logan,  R.  S.  A  survey  analysis  of  military  computer  based  systems:  A 
two  part  study.  Volume  I:  A  survey  and  annotated  bibliography  of 
aids  for  instructional  systems  development  (Report  No.  MDC  E1570). 
Arlington,  VA:  Defense  Advanced  Research  Projects  Agency,  1977. 

(AD  406  902) 

Orlansky,  J.,  &  String,  J.  Cost  effectiveness  of  computer  based  instruc¬ 
tion  in  military  training.  Alexandria,  VA:  Institute  for  Defense 
Analysis,  1979. 

Schulz,  R.  E.,  Hibbits,  N.,  Wagner,  H.,  &  Seidel,  R.  J.  On-line  author¬ 
ing  aids  for  instructional  design  (Technical  Paper).  Alexandria, 

VA:  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social 
Sciences,  in  press. 

Seidel,  R.  J.,  &  Wagner,  H.  A  cost-effectiveness  specification.  In 
H.  F.  O'Neil,  Jr.  (Ed.),  Procedures  for  instructional  systems 
development .  New  York:  Academic  Press,  1979. 

Training  and  Doctrine  Command.  Interservice  procedures  for  instruc¬ 
tional  systems  development  (5  vols.)  (TRADOC  Pamphlet  350-30). 

Fort  Benning,  GA:  Combat  Arms  Training  Board,  August  1975. 

United  States  Continental  Army  Command.  Systems  engineering  of  training 
(CONARC  Reg  350-100-1).  Fort  Monroe,  VA:  Headquarters  United 
States  Continental  Army  Command,  February  1968. 


RESULTS 

AN  ANNOTATED  BIBLIOGRAPHY 
Purpose  of  Block  1.1  Analyze  Job 

To  determine  which  activities  compose  a  job,  what  level  of  perform¬ 
ance  is  adequate,  tinder  what  conditions  the  job  is  done  and  in  what 
order. 

References 

P  1.1.1  Ammerman,  H.  L.  Stating  the  tasks  of  the  job  (Vol.  2). 

Performance  content  for  job  training.  Columbus:  The 
Center  for  Vocational  Education,  The  Ohio  state  Uni¬ 
versity,  1977. 

Procedures  are  given  for  conducting  a  task  inventory. 

P  1.1.2  Christal,  R.  E.  The  United  States  Air  Force  Occupational 
Research  Project  ( AFHRL-TR-73-75) .  Lackland  Air  Force 
Base,  TX :  Air  Force  Human  Resources  Laboratory,  Oc¬ 
cupational  Research  Division,  January  1974.  (AD  774  574) 

Christal  reviews  the  development  and  progress  of  the  Air  Force  Occu¬ 
pational  Research  Project  which  began  in  1958.  The  project  has  pro¬ 
duced  the  Comprehensive  Occupational  Data  Analysis  Programs  package 
(CODAP).  CODAP  is  a  computer-based  system  which  inputs,  reduces,  and 
analyzes  raw  data  from  job  inventories.  It  also  generates  reports 
identifying  individual  and  group  job  characteristics  and  between-job 
differences.  It  is  currently  used  by  all  of  the  Armed  Forces. 

T  1.1.3  Control  Data  Corporation.  Instructional  task  analysis 

(Pub.  No.  76360712).  Minneapolis,  MN:  Control  Data 
Data  Corporation,  1978. 

This  document  reviews  three  techniques  for  task  analysis— (1)  job  task 
analysis,  (2)  information-processing  task  analysis,  and  (3)  hierarchi¬ 
cal  task  analysis. 

P  1.1.4  Folley,  J.  D.  Jr.  Development  of  an  improved  method  of 
task  analysis  (Tech.  Report).  Valencia,  PA:  Ap¬ 
plied  Science  Associates,  June  1964.  (AD  445  869) 

Folley  provides  a  detailed  procedure  for  task  analysis  indicating  what 
information  is  needed  and  how  it  should  be  obtained  and  recorded. 

T  1.1.5  McCormick,  E.  J.,  De  Nisi,  A.  S.  &  Shaw,  J.  B.  The  use 
of  the  position  analysis  questionnaire  (PAQ)  for  es¬ 
tablishing  the  job  component  validity  of  tests  (Report 
No.  5).  West  Lafayette,  IN:  Purdue  University,  1977. 

(AD  A042  270) 
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A  research  program  is  discussed  which  used  the  PAQ,  a  structured  job 
analysis  technique  to  establish  “job  component  validity"  of  tests.  The 
PAQ  is  a  complicated  questionnaire  which  enables  the  respondent  to 
analyze  a  job  in  relation  to  187  job  elements  describing  job  related 
behaviors. 

P  1.1.6  Merrill,  P.  F.  Task  analysis— An  information  process¬ 
ing  approach.  National  Society  for  Performance  and 
Instruction  Journal,  1976,  JH5(2)  7-11. 

This  article  reviews  Gagne's  hierarchical  task  analysis  procedure  and 
presents  an  alternative  approach-information  processing  analysis. 

Merrill  suggests  that  two  skills  have  an  hierarchical  relationship  if 
learning  one  skill  is  prerequisite  to  learning  the  other.  An  informa¬ 
tion  processing  relationship  may  be  observed  if  “the  outputs  of  one 
operation  are  required  as  the  inputs  for  another  operation." 

T  1.1.7  Montemerlo,  M.  0.  &  Eddowes,  E.  The  judgmental  nature 
of  task  analysis.  Proceedings  of  the  Human  Factors 
Society  Annual  Meeting.  Detroit,  1978. 

This  paper  examines  the  state  of  the  art  of  task  analysis.  Montemerlo 
and  Enddowes  suggest  that  "task  analysis"  defies  task  analysis.  They 
use  the  term  "proceduralizable"  to  characterize  those  tasks  which  can  be 
described.  Task  analysis  which  is  largely  judgmental  does  not  lend 
itself  to  proceduralization.  Their  argument  is  supported  by  seven  fac¬ 
tors  which  indicate  that  task  analysis  is  an  artistic,  creative,  syner¬ 
gistic,  multi-purpose,  problem  solving,  global,  interpersonal,  political, 
and  cognitive  task.  They  conclude  that  task  analysis  cannot  be  reduced 
to  one  specific  procedure. 

P  1.1.8  Powers,  T.  E.  Selecting  presentation  modes  according  to 
personnel  characteristics  of  users  and  the  nature  of 
job  tasks.  Part  I  job  tasks.  Baltimore,  MD:  Uni¬ 
versity  of  Maryland,  Baltimore  County,  July-Dee.  1977. 

(AD  AO 38  511) 

In  this  volume.  Powers  examines  the  usefulness  of  ten  job  task  cate¬ 
gories  in  an  effort  to  provide  a  uniform  procedure  for  classifying  any 
job  task.  Seven  categories  were  found  to  be  highly  applicable  to  many 
technical  job  tasks.  Other  volumes  examine  a  task  analysis  technique 
which  distinguishes  the  way  technical  information  is  presented  accord¬ 
ing  to  personnel  aptitude  and  category  of  job  task. 

T  1.1.9  Task  Inventory  Exchange.  Brochure .  Columbus,  OH:  The 
Center  for  Vocational  Education,  The  Ohio  State 
University,  1975. 
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The  Task  Inventory  Exchange  (TIE)  is  a  national  clearinghouse  for  occu¬ 
pational  task  inventories  which  are  lists  of  job  tasks.  The  goal  of 
the  TIE  is  to  increase  the  sharing  of  task  inventories  in  order  to 
reduce  duplication. 

P  1.1.10  Training  and  Doctrine  Command.  Job  and  task  analysis 
regulation  ( TRADOC  Reg.  351-4).  Fort  Monroe,  VAs 
Headquarters,  United  States  Army  Training  and  Doc¬ 
trine  Command,  February  1979. 

This  document  defines  Army  policy,  objectives,  procedures,  and  require¬ 
ments  for  the  conduct  of  job  and  task  analysis.  TRADOC  Pamphlet  350-30 
is  the  basis  for  the  regulation. 

P  1.1.11  Training  and  Doctrine  Command.  Job  and  task  analysis 

handbook  (TRADOC  Pamphlet  351-4).  Fort  Monroe,  VA: 
Headquarters,  United  States  Army  Training  and  Doc¬ 
trine  Command,  February  1979. 


The  handbook  details  procedures  for  job  analysis,  task  analysis,  and 
the  selection  of  tasks  for  training.  The  handbook  was  designed  as  a 
reference  for  analysts  and  not  as  a  complete  authoring  aid  to  prepare 
trainees  to  perform  job  and  task  analysis  procedures. 

P  1.1.12  Washington  State  Department  of  Personnel.  Task  analysis 
handbook .  Olympia,  WA:  Washington  State  Department 
of  Personnel,  July  1973.  (PB  226  682) 

This  source  presents  a  detailed  task  analysis  procedure  for  personnel 
analysts  which  includes  examples  of  the  forms  task  analysts  use.  Un¬ 
fortunately  it  falls  short  of  an  authoring  aid  because  it  does  not  step 
a  task  analysis  trainee  through  the  procedures. 

T  1.1.13  Yaney,  J.  P.  A  critical  review  of  the  instructional 
technology  mechanism  of  task  analysis.  Improving 
Human  Performance:  A  Research  Quarterly,  1973,  3{2) , 
64-70. 

Yaney  offers  a  critical  review  of  the  instructional  technology  mechan¬ 
ism  of  task  analysis.  He  discusses  assets  and  liabilities  of  three  task 
analysis  techniques:  (1)  cognitive  analysis,  (2)  master  performance  and 
(3)  field  analysis.  Yaney  presents  an  operational  cost  model  for  task 
analysis.  He  concludes  that  the  costs  typically  outweigh  the  benefits. 


Status 


A  variety  of  techniques  and  procedures  exist  for  task  and  job  analysis 
(Control  Data  Corporation,  1977;  Ammerman,  1977;  Folley,  1964;  and 
Powers,  1977).  The  notion  of  devising  an  inventory  of  job  tasks  is 
particularly  well  developed.  A  clearinghouse  for  the  dissemination  of 
task  inventories  has  been  established  (TIE)  as  well  as  a  computer-based 
system  (CODAP)  to  analyze  data  gathered  by  job  inventories.  The  TRADOC 
Pamphlet  (351-4)  indicates  the  job  and  task  analysis  data  which  must  be 
collected  but  does  not  provide  the  analyst  with  the  rationale  for  making 
the  decisions  involved  in  gathering  this  data.  An  authoring  aid  for  this 
block  is  unavailable. 
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Purpose  of  Block  1.2  Select  Tasks/Functions 

To  determine  the  tasks  to  be  trained  baaed  upon  certain  selection 
criteria  and  resource  availability. 

References 

P  1.2.1  Ammerman,  H.  L.  Identifying  relevant  job  performance 
(Vol.  3).  Performance  Content  for  Job  Training. 

Columbus:  The  Center  for  Vocational  Education.  The 
Ohio  State  University,  1977.  (ED  146  371) 

Ammerman,  H.  L.  &  Essex,  P.  W.  Deriving  performance  re¬ 
quirements  for  training  (Vol.  4).  Performance  content 
for  job  training.  Columbus:  The  Center  for  Vocational 
Education,  The  Ohio  State  University,  1977.  (ED  146  372) 

Volume  3  provides  a  procedure  for  conducting  surveys  to  determine 
relevant  job  tasks.  The  details  offered  on  the  types  of  survey  ques¬ 
tions  to  be  devised  are  useful.  Volume  4  presents  a  task  selection 
procedure  employing  prediction  equations  which  determine  the  frequency 
of  task  performance. 

T  1.2.2  Page,  E.  B. ,  Jarjoura,  D.  &  Konopka,  C.  D.  Curriculum 

design  through  operations  research.  American -Educa¬ 
tional  Research  Journal,  23(1),  1976,  31-49. 

The  authors  present  a  technique  which  calculates  values  for  the  compo¬ 
nents  involved  in  instructional  decision  making.  This  would  make  a  most 
useful  authoring  aid  considering  the  required  activities  of  Block  1.1. 
According  to  Page  et  al.  operations  research  has  rarely  been  used  to 
determine  what  to  teach. 

A  1.2.3  Schulz,  R.  E.  Job  aid  for  select  tasks  for  training 
(HUMRRQ  Draft  Manual).  Alexandria,  VA:  Human 
Resources  Research  Organization,  September  1978. 

Schulz  provides  an  authoring  aid  for  Block  1.2  following  the  ISD  text. 

A  system  of  worksheets  for  the  collection  of  data,  which  supports  selec¬ 
tion  decisions,  appears  to  be  a  good  technique  for  organizing  a  great 
deal  of  task  analysis  input. 

Status 


Page  et  al.  (1976)  offer  a  technique  for  selecting  tasks  for  training. 
Further  development  could  make  this  a  useful  authoring  aid.  Ammerman 
and  Essex  provide  procedures  for  task  selection.  An  authoring  aid  fol¬ 
lowing  the  ISD  text  is  offered  by  Schulz  (1978) 
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Purpose  of  Block  1.3  Construct  Job  Performance  Measures 
To  devise  job  performance  measures  (JPMs)  for  each  task  selected  for 
training.  JPMs  are  tests  which  indicate  how  well  an  individual  can 
perform  a  job. 

References 

T  1.3.1  Ammerman,  H.  L.  &  Essex,  D.  W.  Deriving  performance  re¬ 
quirements  for  training  (Vol.  4).  Performance  content 
for  job  training.  Columbus:  The  Center  for  Vocational 
Education,  Hie  Ohio  State  University,  1977.  (ED  146  372) 

Volume  4  provides  a  brief  introduction  to  an  algorithmic  method  for 
calculating  level  of  task  performance. 

P  1.3.2  Gael,  S.  Employment  test  validation  studies.  JSAS  Cata¬ 
log  of  Selected  Documents  in  Psychology,  1974,  4,  95. 

(Ms.  No.  7  11) 

Gael  discusses  a  procedure  for  the  design  and  selection  of  instruments 
used  to  test  clerical  performance  of  telephone  company  employees. 

A  1.3.3  Schulz,  R.  E.  Job  aid  for  conducting  task  analysis  (HumRRO 
Draft  Manual).  Alexandria,  VA:  Human  Resources  Re¬ 
search  Organization,  September  1978. 

Although  the  title  differs  from  that  of  Block  1.3  the  activities  de¬ 
scribed  by  this  authoring  aid  are  those  of  1.3.  The  aid  provides  a  hand¬ 
book  for  collecting  and  recording  information  which  operationalizes  each 
job  task. 

Status 


A  procedure  exists  for  the  development  of  job  performance  measures  for 
clerical  staff  (Gael,  1974).  It  is  reasonable  to  expect  that  this  pro¬ 
cedure  could  be  applicable  in  other  vocational  settings.  An  authoring 
aid  exists  following  the  procedures  outlined  in  the  ISD  model  (Schulz, 
1978)  . 
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Purpose  of  Block  1.4  Analyze  Existing  Courses 

To  determine  the  degree  to  which  a  course  includes  the  same  tasks  which 
the  command  needs  to  teach. 

References 

A  1.4.1  Schulz,  R.  E.  Job  aid  for  analyzing  existing  courses 
(HumRRO  Draft  Manual).  Alexandria,  VA:  Human 
Resources  Research  organization,  September  1978. 

Schulz  provides  an  authoring  aid  for  Block  1.4  of  the  ISD  model. 

Status 


An  authoring  aid  exists  for  this  block  which  was  designed  following  the 
guidelines  of  the  ISD  model  (Schulz,  1978).  Procedures  and  techniques 
other  than  those  specified  in  the  ISD  text  itself  were  unavailable.  A 
validation  of  the  Schulz  aid  would  be  particularly  valuable  for  determining 
further  developmental  efforts  for  this  block. 
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Purpose  of  Block  1.5  Select  Instructional  Settings 

To  assign  tasks  to  one  of  the  following  instructional  settings:  (1) 

Job  Performance  Aids,  (2)  Self  Teaching  Exportable  Packages,  (3)  Formal- 
on-the  Job  Training,  (4)  Installation  Support  Schools  or  (5)  Resident 
Schools. 

A  1.5.1  Schulz,  R.  E.  Job  aid  for  selecting  instructional  settings 
(HumRRO  Draft  Manual).  Alexandria,  VA:  Human  Re¬ 
sources  Research  Organization,  September  1978. 

Schulz  provides  an  authoring  aid  which  guides  the  trainee  through  Block 
1.5  of  the  ISD  model. 

Status 


The  purpose  of  this  block  is  unique  to  military  instructional  settings 
which  limits  the  availability  of  references.  Schulz  ( 1978)  has  developed 
an  authoring  aid  covering  the  five  settings  stated  in  the  purpose  of  this 
block.  Techniques  indicating  the  distinctions  and  characteristics  of  the 
five  settings  identified  by  the  model  and  procedures  for  assigning  tasks 
to  all  of  these  settings  would  be  useful.  Authoring  aids  following  other 
instructional  models  would  be  of  interest. 
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Purpose  of  Block  II. 1  Develop  Objectives 

To  develop  learning  objectives  for  the  learning  categories  of  informa¬ 
tion  mental  skills,  physical  skills,  and  attitudes. 

References 

P  2.1.1  Gibbons,  A.  S.  &  Hughes,  J.  A.  A  method  of  deriving  hier¬ 
archies  of  instructional  objectives,  proceedings  of 
the  Human  Factors  Society  Annual  Meeting.  Detroit, 

1978. 

Gibbons  and  Hughes  offer  a  procedure  for  transforming  a  task  list  into 
hierarchies  of  instructional  objectives.  This  document  is  a  detailed 
authoring  procedure  which  utilizes  terminology  of  the  XSD  model.  It 
would  be  a  good  candidate  for  an  authoring  aid  effort. 

P  2.1.2  Hillelsohn,  M.  J.  Instructional  objectives.  Teaching 

with  computers;  A  guide  to  creating  instruction  for 
computer  presentation.  Unpublished  manuscript,  1979. 
(Available  from  Human  Resources  Research  Organization, 
Alexandria,  VA. ) 

Hillelsohn  provides  a  procedure  for  devising  objectives  for  computer- 
based  learning.  With  the  inclusion  of  opportunities  for  practice  this 
document  could  be  readily  modified  into  an  authoring  aid. 

T  2.1.3  Roudabush,  G.  E.  An  empirical  structure  for  reading  ob¬ 
jectives.  A  paper  presented  at  the  annual  meeting 
of  the  American  Educational  Research  Association. 

Chicago,  April  1974.  (ED  095  194) 

Roudabush' s  research  examines  hierarchical  organizations  of  reading  ob¬ 
jectives.  Roudabush  briefly  discusses  instructional  objective  litera¬ 
ture  and  provides  considerable  detail  about  reading  behaviors. 

A  2.1.4  Schulz,  R.  E.  Job  aid  for  developing  objectives  (HumRRO 
Draft  Manual).  Alexandria,  VA:  Human  Resources 
Research  Organization,  December  1978. 

This  document  is  an  authoring  aid  which  follows  the  procedures  outlined 
in  Block  II.I-Develop  Objectives. 
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Status 


The  Gibbons  and  Hughes  (1978)  effort  and  the  Hillelsohn  (1979)  effort 
are  useful  procedures  for  developing  objectives.  The  latter  is  particu¬ 
larly  of  interest  because  it  is  a  procedure  for  devising  objectives  from 
a  task  list  which  is  the  focus  of  this  block.  Both  could  be  modified  to 
form  authoring  aids.  The  Schulz  (1978)  aid  follows  the  ISD  model. 
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DESIGN 


19 


Purpose  of  Block  II .2  Develop  Tests 

To  write  test  items  for  each  learning  objective. 

References 

P  2.2.1  Conoley,  J. ,  &  O'Neil,  H.  F. ,  Jr.  A  primer  for  develop¬ 
ing  test  items.  Zn  H.  F.  O'Neil,  Jr.  (Ed.),  Procedures 
for  instructional  systems  development.  New  York s 
Academic  Press,  1979. 

Conoley  and  O'Neil  provide  detailed  procedures  for  the  construction  of 
multiple-choice  test  items.  Guidelines  for  revisions  are  embedded  in 
test  item  examples. 

P  2.2.2  Gronlund,  N.  Preparing  criterion-referenced  tests  for 
classroom  instruction.  New  York:  Macmillan,  1973. 

Procedures  for  the  development  of  criterion-referenced  tests  are  pro¬ 
vided.  Gronlund  discusses  the  decisions  which  must  be  made  by  the  test 
developers— i.e. ,  the  level  of  mastery  desired,  scope  of  the  subject  mat¬ 
ter  covered,  and  type  of  test  item.  Also  indicated  are  practical  rules 
for  writing  test  items. 

P  2.2.3  Plumlee,  L.  B.  A  short  guide  to  the  development  of  per¬ 
formance  tests  (Professional  Series  Pamphlet  75-1). 

Washington,  DC:  U. S.  Civil  Service  Commission, 

Test  Services  Section  of  the  Personnel  Research  and 
Development  Center,  January  1975. 

Plumlee  provides  an  overview  of  a  procedure  for  development  of  psycho¬ 
motor  tests.  Task  analysis,  test  reliability,  and  validity  are  also  dis¬ 
cussed.  This  source  directs  the  reader  to  more  detailed  references. 

A  2.2.4  Roid,  G.  &  Haladyna,  T.  A  handbook  on  item  writing  for 
criterion-referenced  tests.  San  Diego,  CA:  Navy 
Personnel  Research  and  Development  Center,  1979. 

Roid  and  Haladyna  provide  an  authoring  aid  for  the  development  of  criterion- 
referenced  tests.  The  authoring  aid  has  been  designed  to  be  compatible 
with  the  ISD  model. 

A  2.2.5  Schulz,  R.  E. ,  Hibbits,  N. ,  Wagner,  H.  &  Seidel,  R.  J. 

On-line  authoring  aids  for  Instructional  design 
(Technical  paper).  Alexandria,  VA:  0. S.  Army  Re¬ 
search  Institute  for  the  Behavioral  and  Social 
Sciences,  in  press. 

Schulz  et  al.  provide  an  authoring  aid  for  the  development  of  test  items 
following  Block  II. 2  guidelines. 
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A  2.2.6  Swezey,  R.  W.  &  Pearlstein,  R.  B.  Guidebook  for  develop¬ 
ing  criterion-referenced  teats  (Report  No.  287-AR18(2)- 
IR-0974).  Arlington,  VAi  U.S.  Army  Research  Insti¬ 
tute  for  the  Behavioral  and  Social  Sciences,  August 
1975.  (AD  AO  14  987) 

The  manual  devised  by  Swezey  and  Pearlstein  is  a  well  developed  author¬ 
ing  aid  for  test  construction  in  an  Army  environment. 

Status 


Test  item  development  is  replete  with  procedures  and  authoring  aids  from 
civilian  and  military  environments.  The  Roid  and  Haladyna  (1978)  hand¬ 
book  and  the  Schulz  authoring  aid  have  been  designed  with  attention  to 
the  ISD  model. 
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Purposes  of  Block  II. 3  Describe  Entry  Behavior 

1.  To  develop  and  administer  a  test  of  entry  skills  and  revise  instruct 
tion  according  to  the  results. 

2.  To  devise  a  pretest  which  measures  the  trainee's  current  knowledge 
so  that  instructional  units  mastered  may  be  bypassed. 

References 

No  references  could  be  identified  for  Block  II. 3. 

Status 

The  test  development  aspects  of  this  block  are  most  likely  covered  by 
procedures  and  authoring  aids  identified  in  Block  II. 2.  Procedures  need 
to  be  defined  to  identify  the  trainee's  entry  skills.  Authoring  aids  can 
then  be  developed. 


PHASE  II 
DESIGN 
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Purpose  of  Block  II. 4  Determine  Sequence  and  Structure 

To  devise  a  learning  hierarchy  which  specifies  the  sequence  in  which 

instructional  objectives  must  be  learned. 

References 

P  2.4.1  Airasian,  P.  W.  A  method  for  validating  sequential  in¬ 
structional  hierarchies.  Educational  Technology. 

December  1971,  54-56. 

Airasian  provides  an  explanation  of  the  notion  of  an  instructional  hier¬ 
archy.  He  presents  a  model  for  testing  the  validity  of  an  established 
hierarchy  of  instructional  tasks.  This  procedure  is  needed;  however, 
a  procedure  for  determining  the  interrelationships  between  tasks  is  a 
higher  priority  which  is  not  addressed. 

T  2.4.2  Airasian,  P.  W.  &  Bart,  W.  M.  The  use  of  ordering  theory 
in  the  identification  and  specification  of  instruc¬ 
tional  sequences.  A  paper  presented  at  the  annual 
meeting  of  the  American  Educational  Research  Associa¬ 
tion.  Chicago,  April  1974. 

Airasian  and  Bart  review  a  technique,  which  is  an  outgrowth  of  Guttman, 
for  ordering  instructional  tasks.  As  it  stands  this  document  is  too 
complex.  A  simplified  version  of  ordering  theory  would  be  useful  for 
an  authoring  aid. 

T  2.4.3  Boozer,  R.  F.  &  Lindvall,  C.  M.  An  investigation  of 

selected  procedures  for  the  development  and  evalua¬ 
tion  of  hierarchical  curriculum  structures.  A  paper 
presented  at  the  annual  meeting  of  the  American  Edu¬ 
cational  Research  Association.  February  1971. 

A  study  assessing  the  usefulness  of  Guttman’ s  scalogram  analysis  and  sim¬ 
plex  analysis  for  a  mathematics  program  is  presented.  These  techniques 
provide  procedures  for  investigating  the  sequence  of  instruction  in  terms 
of  student  performance.  Boozer  and  Lindvall  affirm  the  value  of  scalogram 
and  simplex  analysis,  however,  this  report  lacks  sufficient  detail  for 
the  novice  instructional  developer. 

P  2.4.4  Dansereau,  D.  R.,  Evans,  S.  H.,  Wright,  A.  D.,  Long,  G. 

&  Actkinson,  T.  Factors  related  to  developing  in¬ 
structional  information  sequences:  Phase  I  (Report 
No.  AFHRL-TR-73-51( I) ) .  Lowry  Air  Force  Base, 

CO:  Air  Force  Human  Resources  Laboratory,  Technical 
Training  Division,  March  1974.  (AD  777  832) 

Dansereau,  D.  R.,  Evans,  S.  H.,  Actkinson,  T.  A.,  and 
Long,  G.  L.  Factors  relating  to  the  development  of 
optimal  instructional  sequences  ( Report  No.  AFHRI/- 
TR^73-51 (II) ) •  Lowry  Air  Force  Base,  CO:  Air 
Force  Human  Resources  Laboratory,  Technical  Train¬ 
ing  Division,  June  1974.  (AD  783  843) 
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Phase  I  provides  a  review  of  instructional  sequencing  literature.  The 
results  of  an  experiment  conducted  in  Phase  II  indicate  that  a  multi¬ 
dimensional  scaling  procedure  for  generating  instructional  sequences  was 
useful  and  reliable. 

T  2.4.5  Dansereau/  D.  F. ,  Long,  G.  L. ,  Evans,  S.  H.,  6  Actkinson, 

T.  R.  Objective  ordering  of  instructional  material 
using  multidimensional  scaling.  Journal  of  Structural 
Learning,  in  press. 

Dansereau  et  al.  discuss  multidimensional  scaling  as  a  technique  for  de¬ 
vising  instructional  sequences  and  the  interactions  of  sequencing  with 
type  of  material  and  individual  differences.  A  good  review  of  sequenc¬ 
ing  literature  is  provided  but  it  is  insufficient  for  a  procedure  or  an 
authoring  aid. 

P  2.4.6  Durell,  A.  B.  A  computer  simulation  study  of  measures 
for  validating  learning  hierarchies.  A  paper  pre¬ 
sented  at  the  annual  meeting  of  the  American  Educa¬ 
tional  Research  Association,  Chicago,  April  1974. 

Durell  examines  procedures  for  validating  a  hierarchy  of  tasks. 

P  2.4.7  Hillelsohn,  M.  J.  Organizing  instruction.  Teaching  with 
computers;  A  guide  to  creating  instruction  for  com¬ 
puter  presentation.  Unpublished  manuscript,  1979. 

(Available  from  Human  Resources  Research  Organiza¬ 
tion,  Alexandria,  VA.) 

A  procedure  for  structuring  and  sequencing  computer-based  learning  mate¬ 
rials  is  provided. 

T  2.4.8  McDade,  C.  E.  Subsumption  versus  educational  sets  Impli¬ 
cations  for  sequencing  of  instructional  materials. 

Journal  of  Educational  Psychology,  1978,  7£(2),  137- 
141.  (ED  187  983) 

McDade  tested  two  sequences  of  educational  psychology  instruction.  A  good 
review  of  the  literature  on  instructional  sequencing  is  provided. 

P  2.4.9  Phillips,  E.  R.  Development  of  optimal  instructional  se¬ 
quences.  A  paper  presented  at  the  annual  meeting  of 
the  American  Educational  Research  Association,  Chicago, 

April  1974.  (ED  097  374) 

Phillips  reviews  research  supporting  the  notion  that  there  can  be  an  opti¬ 
mal  sequence  of  instruction.  Procedures  for  developing  and  validing  in¬ 
structional  sequences  are  also  presented.  This  is  a  readable  literature 
review  which  could  supplement  the  ISD  text. 
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T  2.4.10 


Reigeluth,  C.  M.  In  search  of  a  better  way  to  organize 
instructions  The  elaboration  theory.  Journal  of 
Instructional  Development,  1979,  2^(3),  8-15. 

Reigeluth  provides  a  brief  introduction  to  the  elaboration  model  of 
sequencing  instruction.  The  model  suggests  that  instruction  should  be 
organized  in  a  "general-to-detailed"  pattern  rather  them  a  hierarchical 
sequence.  A  user's  manual  is  in  press. 

T  2.4.11  Seidl,  N.  W.  &  McKeen,  R.  L.  More  on  the  use  of  student 
generated  learning  hierarchies.  Improving  Human  Per¬ 
formances  A  Research  Quarterly,  1974  J3(2),  71-80. 

Seidl  and  McKeen  examine  learning  hierarchies  for  algebra  instruction 
devised  by  low  and  high  ability  students.  Although  significant  results 
were  not  obtained,  the  notion  of  students  devising  their  own  learning 
hierarchies  deserves  more  attention  in  light  of  current  efforts  to  indi¬ 
vidualize  instruction. 

T  2.4.12  Wellens,  B. ,  Lenke,  J.  M.  &  Oswald,  J.  H.  An  investiga¬ 
tion  of  a  proposed  hierarchy  of  mathematics  concepts 
and  skills  using  ordering  theory.  A  paper  presented 
at  the  annual  meeting  of  the  American  Educational 
Research  Association.  New  York  City,  March  1977. 

The  ordering  theory  approach  to  the  construction  of  instructional  hier¬ 
archies  is  reviewed.  Wellens  et  al.  investigated  the  effects  of  a 
hierarchy  of  mathematical  skills  and  concepts  generated  by  the  order¬ 
ing  theory  technique. 

Status 


Procedures  for  devising  learning  hierarchies  exist  (Dansereau  et  al. , 

1974,  in  press;  Durell,  1974;  Hillelsohn,  1979).  Research  studies  have 
examined  optimal  sequencing  of  instruction  (Phillips,  1974)  and  the  devel¬ 
opment  of  learning  hierarchies  by  students  (Saidl  and  McKeen,  1974).  A 
variety  of  theories  exist  for  sequencing  instruction;  however,  no  reference 
approaches  an  authoring  aid. 
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Purpose  of  Block  III.1  Specify  Learning  Events/Activities 
To  classify  learning  objectives  and  identify  "learning  guidelines 
necessary  for  optimum  learning  to  take  place. " 


References 

T  3.1.1  Aagard,  J.  A.  &  Braby,  R.  Learning  guidelines  and 

algorithms  for  types  of  training  objectives  (TAEG 
Report  No.  23).  Orlando,  FLs  Training  and  Analy¬ 
sis  Group,  March  1976.  (AD  A023  066) 

Aagard  and  Braby  have  identified  11  classes  of  instructional  objectives 
(such  as  rule  learning)  and  have  devised  a  training  strategy  for  each 
class.  The  strategies  specify  learning  events  which  are  compatible 
with  the  instructional  objectives. 

A  3.1.2  Courseware,-  Inc.  Author  training  course.  San  Diego, 

CA:  Courseware,  Inc.,  January  1978.  Also  in  H.  F. 

O'Neil,  Jr.  (Ed.),  Procedures  for  instructional 
systems  development.  New  York:  Academic  Press, 

1979. 

The  Author  Training  Course  is  a  series  of  authoring  aids  designed  to 
assist  course  authors  in  converting  instructional  objectives  into  in¬ 
structional  materials.  The  course  follows  the  guidelines  of  the  ISD 
model.  Particular  attention  is  paid  to  developing  instruction  which 
maximizes  learning. 

Status 

The  Aagard  and  Braby  (1976)  technique  would  make  a  useful  authoring  aid. 
The  second  reference  available  for  this  block  is  a  course  designed  for 
course  authors  who  are  tasked  with  Block  III.1  of  the  ISD  model. 


PHASE  III 
DEVELOP 


III  .1 

Specify 

Learning 

Events/ 

Activities 


Mil  .2 
^Specify 
NInstruction 
^-Management 
Mian  &  Deli¬ 
very  System 


\III .3  \ 

^Review/  N 
NSelect  ' 
►Vxi  sting  s 
\Materia1s' 


M 1 1 .4 

^Develop  ^ 
\InstructionN 


sill. 5 

Mali  date  n 
MnstructionN 


26 


Purpose  of  Block  XIX. 2  Specify  Instructional  Management  Plan  and 
Delivery  System 

1.  To  specify  the  delivery  system  through  media  selection. 


2.  To  prepare  organizational  plan  for  the  instruction  which  indicates 
instructional  activities  of  students  and  instructors. 


References 


P  3.2.1  Braby,  R. ,  Henry,  J.  M. ,  Parrish,  H.  F. ,  &  Swope,  W.  M. 

A  technique  for  choosing  cost-effective  instructional 
delivery  systems  (TAEG  Report  No.  16).  Orlando,  FLs 
Training  and  Analysis  Group,  April  1975,  Revised 
October  1978. 


The  Training  Effectiveness,  Cost  Effectiveness  prediction  Technique 
(TECEP)  is  a  procedure  designed  to  assist  experienced  training  special¬ 
ists  in  the  selection  of  instructional  delivery  systems.  Hie  selection 
is  based  upon  the  media  requirements  of  the  training  objectives  and  the 
cost-effectiveness  of  the  various  delivery  systems  which  meet  these 
requirements  ( see  T  3.1.1). 


P  3.2.2  Bretz,  R.  The  selection  of  appropriate  communication 

media  for  instruction:  A  guide  for  designers  of  Air 
Force  technical  training  programs  (i.eport  No.  R-601-PR). 
Santa  Monica,  CAs  RAND  Corporation,  February  1971. 

(AD  721  443) 


Bretz  discusses  a  procedure  for  media  selection  which  is  based  on  learning 
objectives,  course  content,  student  characteristics,  and  teaching  method. 


T  3.2.3  Department  of  Defense  Audiovisual  Information  System.  A 
user  guide  for  the  audiovisual  products  data  base. 
Arlington,  VAs  Office  of  Information  for  the  Armed 
Forces,  DoD  Directorate  for  Audiovisual  Activities, 
1975. 


This  document  is  a  user  guide  for  a  computer-based  network  of  audio-visual 
products.  The  guide  is  largely  concerned  with  the  manipulation  of  the 
Hazeltine  terminal  which  is  the  delivery  mechanism  for  the  system. 


T  3.2.4  Keeler,  F.  L.  &  Rizzo,  W.  A.  An  evaluation  of  microfiche 
reader  types  for  use  with  programmed  instruction  (TAEG 
Report  No.  35).  Orlando,  FL:  Training  Analysis  and 
Evaluation  Group,  August  1976.  (AD  A029  714) 
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One  study  compared  several  types  of  microfiche  readers  and  indexing 
techniques/  in  an  attempt  to  explore  ease  of  reader  usage  with  Navy  re¬ 
cruits.  A  second  study  examined  differences  in  efficiency  with  the  use 
of  microfiche  versus  traditional  paper  copy.  No  significant  differences 
in  performance  were  observed.  The  studies  answered  important  questions 
concerning  microfiche  media  but  did  not  approach  an  authoring  procedure 
or  aid. 

P  3.2.5  Lindahl/  W.  H.  &  Lin,  B.  W.  An  heuristic  approach  for 

the  scheduling  of  Navy  Training  Courses  (TAEG  Report 
No.  52) •  Orlando,  FL:  Training  Analysis  and  Evalua¬ 
tion  Group,  December  1977.  (AD  A048  183) 

A  procedure  indicating  steps  and  sources  for  scheduling  Navy  training 
sources  is  presented.  The  advantages  of  automated  scheduling  are 
examined. 

T  3.2.6  Lonigro,  J.  F.  &  Eschenbrenner ,  A.  J.,  Jr.  A  model  for 

selecting  the  media  in  technical/vocational  education. 

In  H.  Hitchens  (Ed.),  Selecting  media  for  learning: 
Readings  from  audiovisual  instruction.  Washington, 

D.C.:  Association  for  Educational  Communications  and 
Technology,  1974. 

Lonigro  and  Eschenbrenner  propose  a  model  for  media  selection  which  con¬ 
siders  types  of  learning  required  by  task,  desired  level  of  mastery, 
matches  between  types  of  learning  and  media,  and  media  production  costs. 

T  3.2.7  Ruefli,  T.  W.,  Ruefli,  J.  C.,  O'Neil,  H.,  Jr.,  &  Smith, 

C.  M.  A  scheduling  strategy  for  computer-managed  instruc¬ 
tional  systems.  Socio-Economic  Planning  Science,  1978, 
145-152. 

Ruefli  et  al.  describe  a  technique  for  and  a  test  of  a  scheduling  sys¬ 
tem  for  computer-managed  Air  Force  training.  The  scheduling  system 
assigns  the  student  to  the  next  lesson  based  on  previous  performance 
and  computer  terminal  availability. 

A  3.2.8  Schulz,  R.  E.  Job  aid  for  specifying  instructional  man¬ 
agement  plan  and  delivery  system  (HumRRO  Draft  Manual). 
Alexandria,  VA:  Human  Resources  Research  Organization, 
January  1979. 

Schulz  has  devised  an  authoring  aid  which  guides  an  author  through  the 
procedures  specified  by  Block  III. 2  of  the  ISD  model. 


P  3.2.9  Smith,  E.  H.  Quality  assurance  of  media  devices  and 

courseware  (AFHRL-TR-75-37) .  Lowry  Air  Force  Base, 
CO:  Technical  Training  Division,  Air  Force  Human 
Resources  Laboratory,  October  1975.  (AD  A018  867) 


Smith  presents  procedures  which  assure  quality  in  the  development  of 
visuals  and  recordings.  Procedures  to  select  cassettes  and  films  are 
briefly  discussed.  Smith  focuses  on  specifics  of  good  cassette  and 
film  production. 

T  3.2.10  Spangenberg,  R.  W.  Overview  of  mediated  courseware  for 
learning  centers  (AFHRL-TR-76-37).  Lowry  Air  Force 
Base,  CO:  Technical  Training  Division,  Air  Force 
Human  Resources  Laboratory,  June  1976.  (AD  A033  304) 


Spangenberg  presents  guidelines  for  media  selection  based  on  the  type 
of  information  which  is  to  be  learned  (i.e.,  intellectual  skill,  cogni¬ 
tive  strategy,  verbal  information,  motor  skill,  attitude).  An  effort 
to  rewrite  this  document  following  ISD  terminology  would  make  it  a 
useful  procedure  or  authoring  aid. 

A  3.2.11  Stolovitch,  H.  D.  Systematically  selecting  media:  A 

mathematical  procedure.  HSPI  Journal,  1977,  16(7) , 


Stolovitch  provides  an  authoring  aid  for  devising  "your  own"  media/media 
attribute  matrix  for  media  selection.  A  mathematical  analysis  of  a  com¬ 
pleted  matrix  provides  input  for  decision-making  which  incorporates  ( 1 ) 
the  range  of  media  available  and  (2)  the  media  characteristics  required 
for  attaining  training  goals. 

Status 

Procedures  and  techniques  exist  for  the  selection  of  media  based  on  learner 
characteristics  (Bretz,  1971)  and  type  of  learning  required  by  the  task 
(Lonigro  &  Eschenbrenner,  1974;  Spangenberg,  1976;  Braby  et  al. ,  1978). 
Several  documents  examine  the  use  of  different  types  of  audiovisual  media 
in  depth  including  cassettes,  films,  and  microfiche.  An  authoring  aid 
exists  for  media  selection  which  considers  type  of  media  and  desired 
learning  outcome  (Stolovitch,  1977).  Another  intention  of  Block  III. 2 
is  to  devise  a  schedule  for  the  progress  of  instruction.  A  procedure 
and  a  technique  exist  which  provide  guidance  on  scheduling  traditional 
and  computer-based  courses  (Lindahl  &  Lin,  1971;  Ruefli  et  al. ,  1978). 

The  Schulz  ( 1979)  authoring  aid  covers  media  selection  and  the  prepara¬ 
tion  of  an  organizational  plan. 
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-  V 


Purpose  of  Block  III. 3  Review/ Select  Existing  Materials 
To  review  existing  instructional  materials  and  examine  the  corres¬ 
pondence  with  instructional  objectives  of  the  current  training  need. 

References 

P  3.3.1  Merrill,  M.  D.y  Reigeluth,  C.  M.  &  Faust,  G.  W.  The 

instructional  quality  profile.  In  H.  F.  O’Neil,  Jr. 

( Ed. )  Procedures  for  instructional  system  develop¬ 
ment.  New  York:  Academic  Press,  1979. 

Merrill  et  al.  discuss  a  procedure  for  diagnosing  and  correcting  weak¬ 
nesses  in  existing  instruction  through  the  examination  of  instructional 
objectives.  An  authoring  aid  for  this  procedure  is  in  production 
(P  3.4.11,  P  5.3.1) . 

A  3.3.2  Schulz,  R.  E.  Job  aid  for, review  and  selection  of 

existing  material  ( HumRRO  Draft  Manual).  Alexandria, 

VA:  Human  Resources  Research  Organization,  October 

1978. 

Schulz  has  prepared  an  authoring  aid  for  Block  III. 3  of  the  ISD  model 
which  follows  the  procedures  outlined  in  the  ISD  text. 


Status 


The  authoring  aid  for  this  block,  which  is  now  in  production  (P  5.3.1) 
will  provide  guidance  in  the  analysis  of  existing  instruction.  The 
authoring  Aid  provided  by  Schulz  (1978)  follows  the  procedures  of  the 
ISD  model. 
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Purpose  of  Block  III. 4  Develop  Instruction 

To  prepare  new  instructional  materials  for  learning  objectives  which 
are  not  covered  by  existing  materials. 

References 

A  3.4.1  Bell,  N.  T.  &  Abedor,  A.  J.  Developing  audio-visual  in¬ 
structional  modules  for  vocational  and  technical 
training.  Englewood  Cliffs,  NJ:  Educational  Tech¬ 
nology  Publications,  1977. 

Bell  and  Abedor  provide  an  authoring  aid  for  the  development  of  self- 
instructional  audio-visual  materials  based  on  specified  instructional 
objectives. 

A  3.4.2  Braby,  R. ,  Parrish,  W.  F. ,  Guitard,  C.  R.  &  Aagard,  <7.  A. 

Computer-aided  authoring  of  programmed  instruction 
for  teaching  (TAEG  Report  No.  58).  Orlando,  FL: 

Training  Analysis  and  Evaluation  Group,  June  1978. 

This  report  describes  a  computer  program  (AUTHOR)  that  generates  pro¬ 
grammed  training  manuals  which  teach  symbol  recognition.  The  present 
document  details  computerized  authoring  system  which  produces  a  learn- 
package  containing  a  pre-test,  drills,  and  criterion  tests  based  on 
author  input  of  a  set  of  symbols.  The  next  two  sources  are  companion 
documents  to  A  3.4.2  and  therefore  do  not  have  reference  numbers. 

Guitard,  C.  R.  Computer-aided  authoring  system  (AUTHOR) 
users  guide,  volume  I  (TAEG  Report  No.  59).  Orlando, 

FL:  Training  Analysis  and  Evaluation  Group.  June 
1978. 

This  document  is  an  operator's  manual  which  accompanies  the  previous 
reference.  It  also  lists  the  computer  program  for  AUTHOR  which  assists 
in  the  preparation  of  training  materials. 

Braby ,  R.  Computer-aided  authoring  of  instructional 

materials  (Tech.  Note  2-79).  Orlando,  FL:  Training 
Analysis  and  Evaluation  Group,  June  1978. 

Braby  revi  .ws  the  computer-aided  authoring  technique  presented  in  TAEG 
Reports  No.  58  and  59.  He  also  reports  that  materials  produced  by  com¬ 
puter-aided  authoring  routines  were  found  to  produce  performance  superior 
to  those  using  traditionally  developed  materials. 

A  3.4.3  Braby,  R. ,  Kincaid,  J.  P. ,  &  Aagard,  J.  A.  Use  of 
mnemonics  in  training  materials:  A  guide  for 
technical  writers  (TAEG  Report  No.  60).  Orlando, 

FL:  Training  Analysis  and  Evaluation  Group, 

July  1978. 
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Braby  et  al.  provide  an  authoring  aid  for  writing  mnemonics  (memory  aids 
to  be  included  in  training  materials  for  classroom  instruction.  Nine 
mnemonic  techniques  are  reviewed  and  details  on  when  each  technique 
should  be  used  are  given.  The  following  document  is  a  companion  docu¬ 
ment  to  A  3.4.3. 

Ainsworth,  J.  S.  Symbol  learning  in  Navy  technical 
training  (TAEG  Report  No.  66).  Orlando,  FL: 

Training  Analysis  and  Evaluation  Group,  January 
1979. 

This  report  is  an  evaluation  of  the  technique  developed  by  TAEG  for 
writing  mnemonics  (Report  No.  60)  for  symbol  learning  and  a  feasibility 
demonstration  of  the  computer-based  authoring  system  (A  3.4.2).  Hand¬ 
books  produced  by  the  computer-aided  authoring  routines  were  found  to 
be  desirable  as  well  as  the  complete  algorithm  for  symbol  learning 
during  morse  code  training. 

A  3.4.4  Courseware,  Inc.  Author  training  course  (Data  Item  No. 

0002AD) .  San  Diego,  CA:  Courseware,  Inc.,  January 
1978.  Also  in  H.  F.  O'Neil,  Jr.  (Ed.),  Procedures 
for  instructional  systems  development.  New  York : 

Academic  Press,  1979. 

The  Author  Training  Course  is  a  series  of  authoring  aids  designed  to 
guide  course  authors  through  Block  III. 4  of  the  ISD  model  (A  3.1.2). 

The  goal  of  the  course  is  to  have  authors  develop  instruction  for 
specified  instructional  objectives. 

A  3.4.5  Deterline,  W.  A.  S  Lenn,  P.  D.  Coordinated  instructional 
systems.  Palo  Alto,  CA:  Sound  Education,  1972. 

This  is  a  lesson  book  which  forms  a  self  study  course  on  instructional 
design.  The  book  is  a  thorough  authoring  aid  which  guides  the  author 
through  the  development  of  two  lessons.  Deterline  and  Lenn  indicate 
the  time  needed  to  reach  mastery  in  their  course  of  instructional 
design  is  an  average  of  35  hours. 

P  3.4.6  Hillelsohn,  M.  J.  Production  of  CBLM.  Teaching  with 

computers:  A  guide  to  creating  instruction  for  com¬ 
puter  presentation.  Unpublished  manuscript,  1979. 
(Available  from  Human  Resources  Research  Organization, 
Alexandria,  VA. ) 

Hillelsohn  presents  a  procedure  for  putting  instructional  materials 
into  a  computer  system.  The  steps  involved  in  programming,  inputting, 
and  debugging  the  system  are  detailed. 

P  3.4.7  Joyce,  R.  P. ,  Cheznoff,  A.  P.,  Mulligan,  J.  F.  &  Mallory, 

W.  J.  Fully  procedural! zed  job  performance  aids: 

Handbook  for  JPA  developers  (Tech.  Report).  Vallencia, 

PA:  Applied  Science  Associates,  December  1973. 


Joyce,  R.  P.,  Cheznoff,  A.  P.,  Mulligan,  J.  F.  &  Mallory, 

W.  J.  Fully  procedurallzed  job  performance  aids: 

Handbook  for  JPA  managers  and  training  specialists 
(Tech.  Report).  Vallencia,  PA:  Applied  Science 
Associates,  December  1973.  (AD  775  706) 

Joyce  et  al.  provide  detailed  procedures  for  the  development  of  Job  Guides 
and  Troubleshooting  Aids  for  Air  Force  equipment  maintenance.  The  devel¬ 
opment  of  this  information  into  an  authoring  aid  would  give  input  for 
those  tasked  with  Job  Performance  Aid  development. 

A  3.4.8  Kern,  R.  P.,  Sticht,  T.  G.,  Welty,  D.  S  Hauke,  R.  N. 

Guidebook  for  the  development  of  Army  training  liter¬ 
ature.  Alexandria,  VA:  Human  Resources  Research 
Organization,  November  1976.  (AD  A033  935) 

This  document  is  an  authoring  aid  for  the  development  of  training 
manuals.  The  guide  presents  examples  of  training  manual  problems  in  a 
"before  and  after  revision"  format. 

A  3.4.9  Schulz,  R.  E.,  Hibbits,  N.,  Wagner,  H.  &  Seidel,  R.  J. 

On-line  authoring  aids  for  instructional  design  (Tech¬ 
nical  paper).  Alexandria,  VA:  U.S.  Army  Research 
Institute  for  the  Behavioral  and  Social  Sciences,  in 
press. 

An  authoring  aid  is  provided  for  Block  III. 4  of  the  ISD  manual. 

A  3.4.10  Shriver,  E.  L.  Fully  proceduralized  job  performance  aids: 

guidance  for  performing  behavioral  analyses  of  tasks 
( AFHRL-TR-75-38) .  Lackland  Air  Force  Base,  TX:  Air 
Force  Human  Resources  Laboratory,  Occupational  Research 
Division,  June  1975.  (AD  A015  059) 

Shriver  amends  the  Joyce  et  al.  (see  P  3.4.7)  documents  with  finer  details 
for  job  performance  aid  development. 

P  3.4.11  Wulfeck,  W.  H.,  Ellis,  J.  A.,  Richards,  R.  E.,  Wood, 

N.  E.  &  Merrill,  M.  D.  The  instructional  quality 
inventory:  I.  Introduction  and  overview  (Special 
Report  No.  79-3).  San  Diego,  CA:  Navy  Personnel 
Research  and  Development  Center,  November  1978. 

The  Instructional  Quality  Index  (IQI)  is  a  procedure  for  the  development 
of  instruction  based  on  the  learning  behaviors  and  outcomes  specified  by 
the  instructional  objectives.  Authoring  aids  for  the  IQI  are  in  prepara¬ 
tion.  The  IQI  was  originally  called  the  Instructional  Strategy  Diagnostic 
Profile  (ISDP)  (P  3.3.1,  P  5.3.1). 
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Status 


This  block  is  the  best  developed  in  terms  of  the  availability  of  author¬ 
ing  aids  for  print  and  audio-visual  using  the  ISD  model  (Courseware,  1978; 
Schulz  et  al. ,  1977;  Wulfeck  et  al. ,  1978;  Braby  et  al.  1978)  perhaps 
because  it  is  the  most  basic  to  the  intent  of  the  model.  Authoring  aids 
are  available  for  the  programming  (Hillelsohn,  in  press)  and  are  unavail¬ 
able  for  the  development  of  computer-based  instructional  materials. 


Purpose  of  Block  III. 5  Validate  Instruction 

To  debug  instructional  materials  by  trial  with  students. 

References 

P  3.5.1  Dick,  W.  Applications  of  formative  evaluation  to  the 

instructional  design  process.  A  paper  presented  at 
the  annual  meeting  of  the  American  Educational  Re¬ 
search  Association.  New  York  City,  April  1977. 

(ED  132  135) 

Dick  reviews  procedures  for  formative  evaluation  which  must  be  part  of 
an  instructional  development  plan  so  that  evaluative  information  is 
gathered  during  each  phase  of  the  developmental  process. 

P  3.5.2  Kandaswamy,  S.,  Stolovitch,  H.  D.  S  Thiagarajan,  S. 

Learner  verification  and  revision:  An  experimental 
comparison  of  two  methods.  AV  Communication  Review, 

1976,  24(3),  316-328. 

Kandaswamy  et  al.  examined  the  cost  effectiveness  of  two  procedures  for 
testing  instructional  materials  on  students  and  making  revisions  based 
on  their  feedback.  Instructional  materials  can  be  piloted  with  one  stu¬ 
dent  or  with  a  group  of  students.  Their  research  findings  indicated  no 
significant  difference.  These  results  indicate  that  the  more  cost  ef¬ 
fective  method  would  be  trials  with  one  student  at  a  time. 

T  3.5.3  Rayner,  G.  T.  An  empirical  study  of  a  methodology  for 
the  revision  of  systematically  designed  educational 
materials  (CAI  Center  Tech  Report  No.  24).  Talla¬ 
hassee,  FL:  Florida  State  University,  March  1972. 

(ED  067  877) 

Rayner  developed  and  tested  a  model  which  identified  revisions  for  in¬ 
structional  materials  based  on  content  and  procedural  deficiencies. 

The  model  was  found  to  be  empirically  useful  for  revising  a  health  edu¬ 
cation  course,  (see  Block  V.3.) 

A  3.5.4  Schulz,  R.  E.  Job  aid  for  validating  instruction  (HumRRO 
Draft  Manual).  Alexandria,  VA:  Human  Resources  Re¬ 
search  Organization,  October  1978. 

Schulz  provides  a  detailed  authoring  aid  which  steps  the  trainee  through 
validation  procedures  identified  by  the  ISD  model. 

P  3.5.5  Wulfeck,  W.  H.,  Ellis,  J.  A.,  Richards,  R.  E.,  Wood,  N.  E., 

&  Merrill,  M.  D.  The  Instructional  quality  inventory: 

I.  Introduction  and  overview  (Special  Report  No.  79-3). 
San  Diego,  CA:  Navy  Personnel  Research  and  Development 
Center,  November  1978. 
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IP 


The  IQI  is  a  quality  control  procedure  designed  to  evaluate  instructional 
objectives,  tests  and  materials  designed  by  the  ISD  method  (P  3.3.1, 

P  5.3.1). 

Status 


Procedures,  techniques,  and  authoring  aids  following  the  ISD  model  (Schulz, 
1978;  Wulfeck  et  al. ,  1978)  exist  for  pilot  testing  newly  developed  instruc¬ 
tional  materials. 
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Purpose  of  Block  IV. 1  Implement  Instructional  Management  Plan 
To  initiate  plan  which  specifies  trainees,  learning  activities,  and  re¬ 
sources  (this  is  achieved  by  making  and  completing  checklists). 

References 

The  authors  could  identify  no  references  for  this  block. 

Status 

Some  references  included  in  Block  III. 2  discuss  implementation.  An 
authoring  aid  would  be  useful  for  activities  such  as  the  development  of 
checklist  used  during  implementation. 
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Purpose  of  Block  IV. 2  Conduct  Instruction 

For  the  instructor  to  initiate  the  instruction  using  the  setting,  methods 
and  media  specified  by  the  management  plan. 

References 

P  4.2.1  Hillelsohn,  M.  J.  Inqolementation  of  CBLM.  Teaching 

with  computers:  A  guide  to  creating  instruction  for 
computer  presentation.  Unpublished  manuscript,  1979. 
(Available  from  Human  Resources  Research  Organization, 
Alexandria,  VA. ) 

Hillelsohn  provides  a  procedure  for  initiating  computer-based  instruc¬ 
tion  by  detailing  the  tasks  of  the  course  administrator. 

Status 


The  Hillelsohn  (1979)  procedure  would  make  a  useful  authoring  aid  for 
computer-based  instruction.  Procedures  and  authoring  aids  are  needed 
which  focus  on  the  implementation  of  traditional  classroom  instruction 
utilizing  a  variety  of  media. 
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Purpose  of  Block  V. 1  Conduct  Internal  Evaluation 
1.  To  determine  student  progress . 


2.  To  improve  quality  of  instruction. 

These  goals  are  achieved  through  the  collection  of  data  from  students, 
instructors,  and  supervisors  and  the  evaluation  of  this  data.  Data 
must  indicate  the  relationship  between  student  entry  behavior  and 
achievement  on  the  terminal  learning  objectives.  The  ISD  effort  is  eval¬ 
uated  as  well.  The  development  of  the  instruction  is  monitored  to  see 
that  it  is  consistent  with  prescribed  ISD  procedures. 

References 

p  5.1.1  Borich,  G.  D.  A  systems  approach  to  the  evaluation  of 
training.  In  H.  F.  O'Neil,  Jr.  (Ed.),  Procedures 
for  instructional  systems  development.  New  York: 

Academic  Press,  1979. 

Structured  decomposition  is  a  program  evaluation  procedure  in  which  data 
is  collected  in  each  component  of  a  program.  Discrepancies  observed 
between  planned  and  actual  outcomes  are  examined. 

P  5.1.2  Champagne,  A.  B.  &  Klopfer,  L.  E.  Formative  evaluation 

in  science  curriculum.  Journal  of  Research  in  Science 
Teaching.  1974,  J1(3),  185-203.  (ED  088  718) 

A  procedure  for  the  formative  evaluation  of  a  new  science  curriculum  is 
presented.  Sources  of  evaluation  data  considered  are  the  curriculum  de¬ 
velopment  plan,  student  instructional  materials,  student  behaviors  with 
respect  to  science  content,  student  behaviors  in  the  classroom,  the  teacher 
and  course  marketability. 

P  5.1.3  Elsbree,  A.  R.  &  Howe,  C.  An  evaluation  of  training  in 
three  acts.  Training  and  Development  Journal,  1977, 

(July),  1-19. 

Elsbree  and  Howe  discuss  procedures  for  the  evaluation  of  training  which 
focus  on  (1)  the  establishment  of  the  general  scope  and  objectives  of 
the  evaluation,  (2)  the  production  of  a  blueprint  for  conducting  the 
project  and  (3)  obtaining  data,  interpreting  data,  and  reporting  to  the 
client.  Flow  charts  of  each  of  these  phases  are  provided  as  well  as  a 
narrative  providing  an  .example  specific  to  Civil  Service  training. 

T  5.1.4  Hall,  E.  R.,  Lam,  K.,  &  Be Homy,  S.  Training  effective¬ 
ness  assessment:  Volume  I,  Current  military  training 
evaluation  programs  (TAEG  Report  No.  39).  Orlando, 

FL:  Training  Analysis  and  Evaluation  Group, 

December  1976.  (AD  AO 36  517) 


Hall,  E.  R. ,  Rankin,  W.  C.  &  Aagard,  J.  A.  Training  ef- 
fectiveness  assessment:  Volume  II,  Problems,  concepts 


and  evaluation  alternatives  (TAEG  Report  No.  39). 


Orlando,  PL:  Training  Analysis  and  Evaluation  Group, 
December  1976.  (AD  A036  518) 

Voltsne  I  examines  interservice  training  evaluation  efforts.  Volume  II 
discusses  problems  which  have  been  observed  in  Navy  training  evaluation 
efforts.  A  variety  of  evaluation  techniques  are  reviewed  with  guide¬ 
lines  indicating  the  most  appropriate  use  of  each. 

P  5.1.5  U.S.  Civil  Service  Commission.  Training  evaluation:  A 

guide  to  its  planning,  development,  and  use  in  agency 
training  courses  (Pamphlet  T-13).  Washington,  DC: 

Bureau  of  Training,  U.S.  Civil  Service  Commission, 

May  1971. 

This  pamphlet  describes  a  procedure  for  assessing  effectiveness  of 
internally  developed  and  administered  training  courses. 

P  5.1.6  U.S.  Civil  Service  Commission.  A  process  for  the  evalua¬ 
tion  of  training.  Washington,  DC:  Training  Leader¬ 
ship  Division,  U.S.  Civil  Service  Commission,  April 
1978. 

This  document  provides  a  procedure  for  formulating  and  implementing  an 
evaluation  plan. 

Status 

Procedures  and  techniques  exist  for  student  assessment  (Champagne  & 
Klopfer,  1974;  Hall  et  al. ,  1976).  Procedures  for  planning  internal 
evaluation  are  also  available  (Borich,  1979;  Elsbree  &  Howe,  1977;  U.S. 
Civil  Service  Commission,  1971,  1974).  Unfortunately  authoring  aids 
and  procedures  which  follow  the  ISD  model  are  unavailable. 
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Purpose  of  Block  V.  2  Conduct  External  Evaluation 
To  determine  whether  trainees  meet  the  needs  of  the  command. 

The  external  evaluation  examines  survey  data  of  trainees  In  the  field, 
job  performance  measures,  and  evaluation  of  supervisors. 

References 

P  5.2.1  Department  of  the  Air  Force.  Training  and  evaluation 

course  reviews  (ATC  Regulation  52-1),  Randolph  Air 
Force  Base,  TX:  Headquarters  Air  Training  Command, 
Department  of  the  Air  Force,  September  1971. 

This  document  is  a  regulation  which  details  procedures  for  internal  and 
external  evaluation.  The  intention  of  the  data  collected  is  to  provide 
feedback  for  ISD  course  improvement. 

P  5.2.2  Pennell,  R. ,  Harris,  D.,  Schwille,  J.  Appraisal  of  Air 

Force  training  course  field  evaluation  system  (AFHRL- 
TRr76-63).  Lowry  Air  Force  Base,  CO:  Technical 
Training  Division,  Air  Force  Human  Resources  Labora¬ 
tory,  October  1976.  (AD  A035  641) 

Pennell  et  al.  examined  the  Air  Force  field  evaluation  procedure  with 
the  intention  of  improving  data  analysis  and  reporting.  Recommendations 
were  made  which  focused  on  the  design  of  the  evaluation  instrument  and 
frequency  of  evaluations. 

P  5.2.3  Walker,  R.  A.  Post-hoc  evaluation  of  S-3A  instructional 
systems  development.  San  Diego:  Courseware,  Inc. , 

1978. 

Walker  reports  on  procedures  used  to  evaluate  a  training  course  for  an 
anti-submarine  warfare  aircraft  ( S-3A)  developed  by  an  ISD  approach. 

Status 


Procedures  exist  for  conducting  an  external  evaluation  for  military 
training  courses.  Authoring  aids  are  needed  which  follow  the  ISD  model. 
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Purpose  of  Block  V.3  Revise  System 

To  revise  the  system  based  on  internal  and  external  evaluation  reports 
and  changes  in  instructional  needs  due  to  new  doctrine,  procedures, 
or  weapons  systems. 

References 

P  5.3.1  Ellis,  J.  A.,  Wulfeck,  W.  H. ,  Merrill,  M.  D. ,  Richards, 

R.  E.,  Schmidt,  R.  V.  &  Wood,  N.  D.  Interim  training 
manual  for  the  instructional  quality  inventory  (NPRDC 
TN  78-5).  San  Diego,  CA:  Navy  Personnel  Research 
and  Development  Center,  February  1978. 

The  Instructional  Quality  Inventory  is  an  analysis  procedure  for  evaluating 
and  revising  instructional  materials.  This  document,  which  is  currently 
undergoing  revision,  is  a  manual  designed  to  train  instructional  developers 
in  the  use  of  the  IQI  (P  3.3.1,  P  3.4.11). 

T  5.3.2  Gropper,  G.  L.  Diagnosis  and  revision  in  the  development 
of  instructional  materials.  Englewood  Cliffs,  NJ: 

Educational  Technology  Publications,  1975. 

Gropper  reviews  types  of  student  failure  and  program  failure  which  indi¬ 
cate  needs  for  program  revision. 

T  5.3.3  Rayner,  G.  T.  An  empirical  study  of  a  methodology  for  the 

revision  of  systematically  designed  educational  materials 
(CAI  Center  Tech  Report  No.  24).  Tallahassee,  FL: 

Florida  state  University,  Marcy  1972.  (ED  067  877) 

Rayner  has  designed  a  model  for  revising  the  content  and  procedures  of 
instruction  which  incorporates  instructional  objectives,  diagnostic  tests, 
prescriptions,  and  posttests.  The  model  has  been  successfully  tested  on 
health  education  courses. 

Status 


A  procedure  and  technique  exist  for  revising  a  course  based  on  internal 
and  external  evaluation  data.  An  authoring  aid  (Ellis  et  al. ,  1978j 
P  3.4.11)  following  the  ISD  model  is  in  preparation.  More  procedures  and 
authoring  aids  would  be  valuable. 
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CONCLUSIONS 


i 


The  following  section  presents  references  for  authoring  systems. 
The  intent  of  these  systems  is  to  provide  authoring  aids  or  procedures 
for  all  steps  of  the  instructional  systems  design  process.  A  summary 
is  then  provided  indicating  the  availability  of  authoring  aids,  pro¬ 
cedures,  and  techniques  by  phase.  Finally  suggestions  for  research 
and  development  are  made  in  light  of  the  availability  of  such  materials 
as  noted  in  the  summary. 


Authoring  Systems 

Several  systems  have  been  developed  or  are  in  development  which 
guide  an  author  through  complete  phases  of  an  instructional  development 
effort.  The  systems  use  a  variety  of  media  for  delivery  and  generally 
incorporate  the  procedures  identified  by  the  ISD  model. 

Control  Data  Corporation.  Computer  resource  for  effective 
author  training  in  education  (CREATE).  Minneapolis, 

MN;  Control  Data  Corporation,  1978. 

CREATE  is  a  curriculum  for  the  development  of  individualized  instruc¬ 
tion.  A  variety  of  media  is  used  to  deliver  the  course  including 
printed  text,  videotapes,  and  CAI.  The  student  author  is  expected  to 
take  the  course  at  a  learning  center  so  that  all  media  are  readily 
accessible.  The  curriculum  is  modularized  so  that  a  program  of 
instruction  can  be  designed  for  each  student. 

Department  of  the  Air  Force.  Training  handbook  for  designers 
of  instructional  systems  (AFP  50-58).  Washington,  DC: 
Department  of  the  Air  Force,  July  1978. 

The  Air  Force  training  handbook  is  a  set  of  authoring  aids  or  proce¬ 
dures  detailing  an  instructional  systems  development  model  ( AFM  50-2). 
The  major  steps  of  the  model  are:  (1)  Analyze  Systems  Requirements; 

(2)  Define  Education  or  Training  Requirements;  (3)  Develop  Objectives 
and  Tests;  (4)  Plan,  Develop,  and  Validate  Instruction;  and  (5)  Conduct 
and  Evaluate  Instruction.  Instructional  developers  receive  guidance 
in  filling  out  a  series  of  forms  for  each  step  of  the  ISD  process. 

Hughes,  J.,  O'Neal,  A.  F.,  Kearsley,  G.,  O'Neal,  H., 

Fee,  M. ,  Rod,i,  L.,  &  Hermanns,  J.  Functional  specifi¬ 
cations  to  computer  aided  training  systems  development 
and  management  ( CATSDM )  support  functions  (Tech.  Report 
N AVTRAEQU I PCEN  71-C-0018-1 ) ,  Orlando,  FL:  Naval 
Training  and  Equipment  Center,  July  1978. 
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This  reference  is  a  specification  for  a  system  of  computer  programs 
which  manages  the  total  ISD  process. 


Harris,  W.  P.  An  authoring  system  for  on-the-job  environ¬ 
ments.  In  H.  F.  O'Neil,  Jr.  (Ed.),  Issues  in  instruc¬ 
tional  systems  development.  New  Yorks  Academic  Press, 
1979. 

The  Lincoln  Laboratory  has  devised  and  field  tested  a  system  which  en¬ 
ables  subject  matter  experts  (SMEs)  to  create  computer-deliverable  on- 
the-job  training  materials.  A  Lincoln  Terminal  System  was  used  to 
deliver  the  authoring  system.  The  authoring  training  lessons  were 
created  to  guide  SMEs  through  the  development  of  CAI  materials  without 
providing  instructional  development  theories. 

Status 


A  system  incorporating  all  blocks  of  the  ISD  model  is  currently  unavail¬ 
able.  The  Department  of  the  Air  Force  Training  Handbook  (1978)  is  not 
directly  applicable  for  Army  training  developers  because  of  its  reliance 
upon  Air  Force  terminology.  The  Hughes  et  al.  (1978)  authoring  support 
system  will  be  most  appropriate  to  this  need  once  it  has  been  developed. 
Throughout  the  prior  reference  section  a  series  of  authoring  aids  pre¬ 
pared  by  Schulz  (Human  Resources  Research  Organization)  have  been  noted. 
The  aids  have  been  developed  under  contract  to  the  U.S.  Army  Research 
Institute  for  Behavioral  and  Social  Sciences.  By  October  of  1979  thir¬ 
teen  aids  covering  Phases  I,  II,  and  III  (excluding  Block  1.1)  will  have 
been  developed  and  evaluated. 


Summary  of  Availability  of  Authoring  Materials  by  ISD  Phase 


1.1 

1.2 

1.3 

1.4 

1.5 

Analyze 

Select 

Construct 

Analyze 

Select 

PHASE  I  Job 

Tasks/ 

Job  Per- 

Existing 

Instruc- 

ANALYZE 

^  Functions 

^  formance 
Measures 

^  Courses 

^  tional 

Setting 

P  T 

apt 

APT 

A 

A 
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Job  information  is  collected  and  analyzed  during  Phase  I.  Tasks 
are  identified  for  training  and  tests  are  developed  to  show  job  mastery. 

An  analysis  of  existing  courses  is  made  to  determine  the  availability 
of  courses  designed  to  teach  the  tasks  selected  for  training.  The  best 
match  between  selected  tasks  and  instructional  setting  is  then  determined.  >•  ■ 

Schulz  (A  1.2.3,  A  1.3.3,  A  1.4.1,  A  1.5.1)  has  devised  authoring 
aids  for  all  of  the  above  activities  except  job  analysis.  Hie  Schulz 
aids  provide  forms  to  be  filled  out  covering  each  procedure  identified 
by  the  ISD  model.  Although  an  authoring  aid  does  not  exist  for  Block 
I.l-Analyze  Job,  there  are  procedures  available  which  could  be  converted 
to  an  authoring  aid  format  (Training  and  Doctrine  Command,  P  1.1.11). 


PHASE  II 
DESIGN 


II. 1 

II. 2 

II. 3 

II. 4 

Develop 

Develop 

Describe 
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Objectives 

APT 

Tests 
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A  P 

Entry 

^  Behavior 

Sequence  & 
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P  T 

The  Design  phase  utilizes  the  job  analysis  data  produced  in  Phase  I. 
Learning  objectives  are  devised  for  each  task  and  test  items  are  pro¬ 
duced  for  each  learning  objective.  Pretests  are  developed  to  examine 
the  trainees'  knowledge  prior  to  instruction  so  that  the  level  of  in¬ 
struction  is  appropriate  to  the  needs  of  its  users.  An  analysis  of 
the  learning  obj  ectives  is  made  to  determine  the  most  appropriate  order 
for  the  instructional  activities. 

Authoring  aids  exist  for  the  development  of  objectives  (Schulz, 

A  2.1.4),  and  test  items  (Roid  &  Haladyna,  A  2.2.4?  Schulz,  A  2.2.5;  Swezey  & 
Pearlstein,  A  2.2.6).  Procedures  are  available  for  sequencing  instruction 
by  creating  learning  hierarchies  (Dansereau  et  al. ,  P  2.4.4,  T  2.4.5; 

Durell,  P  2.4.6)  but  an  authoring  aid  is  not  available.  Neither  techniques, 
procedures,  nor  authoring  aids  are  available  for  Block  II. 3-Describe  Entry 
Behavior. 
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PHASE  III 
DEVELOP 


1 1 1 . 1 

III. 2 
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During  Phase  III  the  learning  objectives  are  classified  so  that  the 
optimal  instructional  events  to  produce  learning  can  be  identified.  A 
plan  is  made  which  specifies  the  allocation  of  training  personnel,  in¬ 
structional  materials  and  plant  requirements  for  the  conduct  of  instruc¬ 
tion.  At  this  time  a  determination  of  the  most  appropriate  instructional 
delivery  system  (considering  a  variety  of  media)  is  made.  A  review  of 
instructional  materials  is  made  to  determine  whether  existing  instruc¬ 
tion  meets  the  requirements  of  the  learning  objectives.  New  instruction 
is  developed  for  objectives  not  taught  by  existing  materials.  A  pilot 
test  is  then  conducted  to  try  out  the  instruction. 

Phase  III  is  the  best  developed  in  terms  of  the  availability  of 
authoring  aids  for  print  and  audio-visual  materials.  At  least  one  aid 
is  available  for  each  block.  Authoring  aids  are  needed  however  for  the 
development  of  computer-based  instructional  materials.  A  considerable 
number  of  procedures  exist  for  media  selection  and  the  development  of 
instruction. 


IV. 1 

IV. 2 

Implement 

Conduct 

PHASE  iv  Instruc- 

Instruction 

IMPLEMENT  „10na 
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Management 

Plan 

The  plan  which  specifies  the  trainees,  learning  activities,  and 
physical  resources  is  initiated  during  Phase  IV.  The  instructor  re¬ 
ceives  an  orientation  to  the  course,  conducts  the  instruction,  and 
records  student  data. 

Authoring  aids  are  unavailable  for  both  blocks  of  Phase  IV.  Pro¬ 
cedures  were  also  unavailable  for  Block  IV. 1.  The  single  procedure 
identified  for  Block  IV. 2  could  be  converted  to  an  authoring  aid; 
however,  its  scope  is  limited  to  computer-based  instruction. 
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PHASE  V 
CONTROL 


V.l 
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During  Phase  V  the  data  collected  about  student  performance  are 
evaluated  to  reveal  deficiencies  in  the  instruction.  On-the-job  per¬ 
formance  of  trainees  is  also  of  interest  to  determine  whether  the  in¬ 
struction  meets  the  needs  of  the  Command.  Finally  the  internal  and 
external  evaluation  data  are  summarized  to  provide  input  for  revisions 
to  be  made  on  the  completed  instructional  system. 

Authoring  aids  are  unavailable  for  the  evaluation  phase.  A  variety 
of  procedures  exist  for  conducting  internal  and  external  evaluations  but 
none  use  the  activities  and  terminology  specified  in  the  ISD  model.  Hie 
procedure  for  revising  instruction  devised  by  Ellis  et  al.  (P  5.3.1)  will 
evolve  into  an  authoring  aid  directly  applicable  to  Block  V.3. 


Summary  and  Plans  for  Research  and  Development 

In  Figure  2  the  availability  of  authoring  aids  for  the  complete 
ISD  model  is  summarized.  Briefly,  sun  authoring  aid  is  needed  for 
Block  I.l-Analyze  Job.  Procedures  exist  for  job  analysis  so  an  author¬ 
ing  aid  could  be  readily  developed.  Block  II. 3  would  require  a  greater 
effort  because  procedures  would  need  to  be  developed  as  well.  Phase  III- 
DEVELOP  has  a  sufficient  number  of  authoring  aids  and  procedures  to 
recommend  no  further  effort  for  print  materials.  However,  authors  aids 
are  required  for  computer-based  instructional  materials.  Phases  IV- 
IMPLEMENT  and  V— CONTROL  need  authoring  aid  development.  Phase  V  would 
be  a  higher  priority  than  Phase  IV  because  quality  control  is  essential 
to  an  instructional  design  effort. 

Three  concluding  comments  regarding  this  state-of-the-art  assessment, 
of  Instructional  Systems  Development  are  salient.  First,  there  exists  a 
considerable  lack  of  information  on  the  best  ways  of  managing  the  instruc¬ 
tional  systems  development  process.  Second,  the  interrelationship  between 
the  ISD  process  and  instructional  strategies  is  an  important  one,  yet  the 
knowledge  and  technology  base  for  understanding  this  relationship  is  lack¬ 
ing.  An  ongoing  state-of-the-art  assessment  in  instructional  strategies 
will  address  this  issue.  Third,  although  most  persons  in  the  field  believe 
that  the  instructional  system  development  process  is  an  expensive  one,  there 
is  a  lack  of  accurate  detailed  ISD  cost  data. 


The  final  product  of  this  research  and  development  effort  is 
viewed  as  an  integrated  instructional  system  composed  of  a  series 
of  field  tested  self-paced  modules.  The  modules  will  be  ISD  authoring 
aids,  an  author  management  system,  and  a  computer-based  instruction 
course  to  teach  how  to  use  both  the  job  aids  and  management  system. 

The  curriculum  will  be  designed  so  that  extensive  resources,  instructor 
personnel  or  facility  support  will  not  be  required.  The  modules  will  be 
available  in  the  form  of  audio-visual  materials,  books,  and  documented 
computer  programs.  Evaluation  data  regarding  its  cost-effectiveness 
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l  Naval  Aerospc  Med  Res  Lab,  Pensacola,  ATTN:  Code  L5 
1  Chief  of  NavPers,  ATTN;  Pers-OR 
I  NAVAIRSTA.  Norfolk.  ATTN:  Safety  Ctr 
I  Nav  Oceanographic,  DC,  ATTN.  Code  6251,  Charts  ft  Tech 
1  Center  of  Naval  Anal,  ATTN.  Doc  Ctr 
\  Nav AirSysCom.  ATTN:  AIR  531 3C 
I  Nav  BuMed.  ATTN:  713 
1  Nav  Helicopter  So  hSqua  2.  FPO  SF  96601 
1  AFHRL  (FT!  Williams  AFB 
1  AFHRL  (TTl  Lowry  AFB 

1  AFHRL  IAS)  WPAFB,  OH 

2  AFHRL  (DOJZ)  Brooks  AFB 

1  AFHRL  IDOJN)  Lackland  AFB 
1  HQUSAF  (INYSD) 

I  HQUSAF  (DPXXA) 

1  AFVTG  <R0)  Randolph  AFB 

3  AMRL  (HE)  WPAFB.  OH 

2  AF  Inst  of  Tech,  WPAFB.  OH.  ATTN:  ENE/5L 
1  ATC  (XPTDI  Randolph  AFB 

1  USAF  AeroMed  Lib.  Brooks  AFB  (SUL  4),  ATTN:  DOC  SEC 
I  AFOSR  (NL).  Arlington 

I  AF  Log  Cmd,  McClellan  AFB.  ATTN:  ALC/DPCR8 

1  Air  Force  Academy,  CO,  ATTN:  Dept  of  Bel  Sen 
5  NavPers  ft  Dev  Ctr,  San  Diego 

2  Navy  Med  Neuropsychiatric  Rsch  Unit,  San  Diego 
1  Nav  Electronic  Lab,  San  Diego,  ATTN:  Res  Lab 

1  Nav  TrngCen,  San  Diego.  ATTN:  Cotie  9000-  Lib 
1  NavPostGraSch.  Montmry,  ATTN:  Code  55Aa 
1  N.ivPostGraSch,  Monterey.  ATTN:  Code  2124 
1  NavTrngEquipCtr,  Orlando.  ATTN:  Tecb  Lib 
1  US  Dept  of  Labor,  DC,  ATTN:  Manpower  Admin 
1  US  Oept  of  Justice,  DC,  ATTN:  Drug  Enforce  Admin 
1  Nat  Bur  of  Standards,  DC,  ATTN:  Computer  Info  Section 
1  Nat  Clearing  House  for  MH-  Info,  Rockville 
t  Denver  Federal  Ctr,  Lakewood,  ATTN:  BLM 
1?  Defense  Documentation  Center 

4  Dir  Psych.  Army  Hq,  Russell  Ofct,  Canberra 

1  Scientific  Advsr,  Mil  Bd,  Army  Hq.  Russell  Ofcs,  Canberra 
1  Mil  and  Air  Attache,  Austrian  Embassy 

1  Cmtlie  d*  Recherche  Des  Facteurs,  Humaine  de  la  Defense 
Natinnale,  Brussels 

2  Canatlian  Join!  Staff  Washington 

1  C/Air  Staff.  Royal  Canadian  AF,  ATTN:  Pars  Std  Anal  Br 

3  Chief.  Canadian  Def  Rsch  Staff,  ATTN:  C/CRDS(W) 

4  Bntish  Del  Staff,  British  Embassy.  Washington 


1  US  Marine  Corps  Liaison  Ofc.  AMC,  Alexandria,  ATTN  AMCGS  -  t 
1  USATRADOC.  Ft  Monroe.  ATTN:  ATRO-ED 

6  USATRADOC,  Ft  Monroe,  ATTN:  ATPR  AD 
1  USATRADOC.  Ft  Monroe.  ATTN:  ATTS  EA 

1  USA  Forces  Cmd,  Ft  McPIierson,  ATTN:  Library 

2  USA  Aviation  Test  Bd.  Ft  Rucker,  ATTN:  STEBG--PO 

I  USA  Agcy  for  Aviation  Safety,  Ft  Rucker,  ATTN:  Library 
1  USA  Agcy  for  Aviation  Safety,  Ft  Rucker,  ATTN:  Educ  Advisor 
1  USA  Aviation  Sch,  Ft  Rucker,  ATTN:  PO  Drawer  0 
1  HQUSA  Aviation  Sys  Cmd.  St  Louis,  ATTN:  AMSAV-ZDR 

7  USA  Aviation  Sys  Test  Act.,  F.eJwords  AFB.  ATTN:  SAVTE  T 
I  USA  Air  Oef  Sch.  Ft  Bliv.,  ATTN:  ATSA  TFM 
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